P01716 

IIP-115-A 

Position Measurement Method, an Apparatus, a Computer Program and a 
Method for Generating Calibration Information 

BACKGROUND OF THE INVENTION 
[001] 5 The present invention relates to a technique which provides 

measurement of the position of an object from an image captured by a camera 
unit. 

[002] Three-dimensional measurement with images has recently been 

prevailing as a result of development of processing for the image of a CCD 

10 camera or a computer. There are examples such as a technique which 
measures the position of an object with plural cameras, and the other one 
which uses a combination of a camera and a floodlight. The former calculates a 
distance from the camera to the object with triangulation incorporating a 
position of each camera and a pixel representative of the object in an image 

15 captured by each camera. On the other hand, the latter calculates it with 
triangulation based on a relative position between the camera and floodlight, 
and a position of a beam of collimated light in an image captured by the 
camera. The Japanese Published Patent Application 9-287927 reports related 
arts associated with the latter technique. In this connection, these techniques 

20 are based on the assumption that a lens system of the camera can be modeled 
by a pin-hole camera. As shown in FIG. 1, a model of pin-hole camera is 
characterized in that only a beam of light coming through a pin-hole H reaches 
a plane of image, so that a three-dimensional space (x, y, z) is translated into a 
two-dimensional space (u, v). The model is thus generated on condition that 

25 the incident beam of light passes through the pin-hole, focusing an image in 
the camera. 
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[003] However, it is known that the image captured by the camera with the 

lens system has intrinsic non-linear distortion, and the further it departs from 
the center, the larger the distortion will be. In this way, it is not possible to 
measure the accurate position of an object from the image having this type of 
5 distortion. A technique for improving the accuracy with compensation for the 
captured image has been proposed. The Japanese Published Patent 
Application 5-274426 reports related arts. 

[004] The related arts described above, which make the compensation for the 

image based on the phenomenon that the peripheral distortion is larger, does 
10 not provide fundamental compensation. For example, selecting a matrix of 
feature points in the center of an image where the distortion is regarded as 
relatively limited, the technique disclosed by Japanese Published Patent 
Application 5-274426 calculates a reference point applied to the whole feature 
points of the image. Because the center is not free from the distortion, the 
is reference point is accordingly calculated from the image having the distortion. 
In this way, although it may be possible for this technique to improve the 
accuracy for measurement of a position to some extent, there still remains a 
problem that the sufficiently accurate position measurement may not 
necessarily be implemented. 

[005] 20 The conventional compensation for the distortion of lens system depends 

on a distance between the camera and object. If the distance is known, it may 
be possible to make compensation with a function of compensation for the 
distortion of image, which varies non-linearly according to the distance. 
However, it has not yet solved the problem because actually the distance is 
25 unknown before the measurement of a position is completed. 
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SUMMARY OF THE INVENTION 
[006] The present invention is directed to overcoming of the problem above 

described. It seeks to provide a position measurement method, which 
fundamentally compensates the distortion of an image captured by a camera 
5 unit with a lens system and measures the accurate position of an object 
accordingly, an apparatus, a computer program and a method for generating 
calibration information to be used for the position measurement method and 
the apparatus. 

[° 07 ] According to an aspect of the present invention, a method is provided, 

10 which is applied to measurement for a position of an object according to an 
image of the object captured by a camera unit. The method includes the 
following steps: calculating a discrepancy of an incident beam of light 
penetrating a lens system of the camera unit relative to an optical center of the 
lens system, and compensating the position of the object according to the 
15 discrepancy. 

[008] The method described above, which calculates the discrepancy of the 

incident beam of light penetrating the lens system with regard to its optical 
center, allows recognition of the displacement of the pixel which the camera 
unit finds representative of the position of the object. In this way, it is possible 
20 to detect the accurate position of the object by making compensation according 
to the displacement of the pixel representative of the position of object with 
regard to the optical center. In this connection, the optical center is meant to 
represent the center of the lens system, corresponding to the position of a pin 
hole in case of a pin-hole camera model. 

[009] 2 5 According to another aspect of the present invention, a method is 

provided, which is applied to measurement for a position of an object with a 
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combination of an image of the object captured by a camera unit and 
calibration information. The calibration information is prepared in advance in 
such a manner that a position of a measurement pixel of the image is 
correlated with a direction of an incident beam of light and a displacement 
from a reference point to the incident beam. The method includes the following 
steps: (a) incorporating the image, (b) detecting a position of a pixel 
representative of the object in the image incorporated at step (a); and (c) 
calculating the position of the object according to the direction and the 
displacement of the incident beam, which are obtained from the calibration 
information with reference to the position of the pixel detected at step (b). 

The method described above, which recognizes the discrepancy of the 
incident beam of light that impinges on the pixel representative of the object 
before detection, can detect the accurate position of the object by making 
compensation according to the discrepancy. 

According to still another aspect of the present invention, an apparatus 
is provided, which measures a position of an object according to an image of 
the object captured by a camera unit. The apparatus includes an image input 
means, a pixel position detection means, a storage means for storing 
calibration information and a position calculation means for calculating the 
position of the object according to the direction and the displacement of the 
incident beam of light. 

The apparatus described above has the image input means which 
incorporates the image of object taken by the camera unit and the pixel 
position detection means which detects the position of the pixel representative 
of the object. The apparatus also has the position calculation means which 
calculates the position of the object according to the direction and the 



P01716 



displacement of the incident beam of light that are derived from the 

calibration information. 
[° 13 1 According to yet another aspect of the present invention, an apparatus 

is provided, in which a camera unit includes cameras in sets of at least two so 
5 as to take a plurality of images and a storage means stores the calibration 

information for each camera. 
[014] The apparatus described above can recognize the discrepancy of the 

incident beam of light impinging on the pixel representative of the object of 

each image before it detects the position of the object according to the plurality 
10 of images because the storage means stores the calibration information for 

each camera. In this way, the apparatus can detect the accurate position of the 

object by compensation according to the discrepancy. 
[015] According to a further aspect of the present invention, an apparatus is 

provided, in which a pixel position detection means detects a position of a pixel 
15 representative of an object that has a marker identifying its typical spot. 
[016] The apparatus described above, in which the pixel position detection 

means detects the position of the pixel according to the marker, can 

concentrate its effort of detection on the marked spot of the object. The 

apparatus can detect the accurate position of the object by compensation 
20 according to the discrepancy because it recognizes the discrepancy of the 

incident beam of light impinging on the pixel representative of the marker 

before it detects the position of the marker in the image. In this connection, it 

may alternatively be possible to select a spot such as an edge of an object 

instead of introducing a marker. 
[017]25 According to a still further aspect of the present invention, an apparatus 

is provided, which further includes a floodlight that projects a collimated beam 
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of light on the object. A camera unit captures an image of the object 
illuminated by the collimated beam, and the apparatus measures the position 
of the object according to spatial relationship between a position of the 
floodlight and an optical center of the camera unit as well as a position of the 
5 collimated beam on the image. 

[018] The floodlight of the apparatus described above projects the collimated 

beam of light on the object. The collimated beam may include a spot light beam, 
a slit light beam in the form of bands and a pattern light beam having a given 
pattern, as long as it projects a beam of light in a not divergent form. It may 
10 also be possible to select a combination of them. 

[019] The image input means incorporates the image captured by the camera 

unit and the pixel position detection means detects the position of the pixel 
which is illuminated by the collimated beam of light in the image. 
Furthermore, accessing to the calibration information, the position calculation 
15 means obtains the direction and displacement of the incident beam of light 
representative of the position of the pixel, so that it calculates the position of 
illumination, namely the position of the object, according to the direction and 
displacement. 

[020] In this way, the apparatus of the present invention can measure the 

20 position of the object according to the accurate direction of the incident beam of 
light penetrating the lens system. 
[021] According to a yet further aspect of the present invention, a computer 

program for a computer used for an apparatus is provided, which generates 
calibration information correlating a position of a measurement pixel of an 
25 image captured by a camera unit with a direction of an incident beam of light 
and a displacement from a reference point to the incident beam of light, and 
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measures a position of an object according to an image of the object captured 
by the camera unit and the calibration information. The computer program 
executes the computer in a process comprising: (a) incorporating the image of 
the object, (b) detecting a position of a pixel representative of the object in the 
image incorporated at process (a), and (c) calculating the position of the object 
according to the direction and the displacement of the incident beam of light, 
which are derived from the calibration information with reference to the 
position of the pixel detected at process (b). 

The computer program described above executes the computer so that 
the image input means can incorporate the image captured by the camera unit 
and the pixel position detection means can detect the position of the pixel 
representative of the object in the image. In this way, the position calculation 
means can obtain the direction and the displacement of the incident beam of 
light according to the position of the pixel and the calibration information, 
thereby calculating the position of the object. 

According to another aspect of the present invention, a method for 
generating calibration information is provided, which includes the following 
steps: projecting a beam of light on individual pixels of a camera image; 
according to the incident beam of light for each pixel, calculating a 
displacement from a reference point to the incident beam of light and 
generating the calibration information by correlating a direction and the 
displacement of the incident beam of light with a position of each pixel. 

The method described above, in which a beam of light is imposed on 
each pixel of the image, specifies the direction of the incident beam of light for 
the pixel. In this connection, the direction of the beam of light may be fixed if 
measurement is conducted for at least two locations of the light source. And 
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the method can provide calculation of a displacement from the reference point 
to the beam of light, thereby allowing generation of the calibration data 
correlating the direction and displacement with the position of the pixel. The 
data has the nature of quantitatively calibrated characteristics of the camera. 

[025] 5 According to still another aspect of the present invention, a method for 

generating calibration information is provided, which includes the following 
steps: adjusting a first direction of a camera unit and measuring a first 
relative position of the light source relative to the camera unit, similarly 
adjusting a second direction of the camera unit and measuring a second 
10 relative position of the light source relative to the camera unit, repeating 
determination of an incident beam of light impinging on the measurement 
pixel according to the first and second relative positions for predetermined 
measurement pixels, calculating a displacement from a reference point to the 
incident beam of light for each of the measurement pixels, and generating the 
is calibration information which correlates a direction and the displacement of 
the incident beam of light with each of the measurement pixels. 

[026] In the method described above, adjustment of the first direction of the 

camera unit is made by controlling its pan and tilt so that the first peak 
intensity of light emitted by the light source falls in the measurement pixel 
20 captured by the camera unit. In this way, the first relative position of the light 
source relative to the camera unit is measured. The relative position measured 
as described above serves as a first position, based on which the incident beam 
of light impinging on the measurement pixel is determined. 

[ 027 1 Similarly, the second relative position of the light source is measured 

25 and serves as a second position. 

[028] In this way, these two relative positions make it possible to determine 
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the incident beam of light impinging on the pixel. 
[029] In the method, a step for determining an incident beam of light is 

repeated according to the number of measurement pixels, and the 
displacement from the reference point to the incident beam is calculated. In 
5 this way, the calibration data is generated, which correlates the direction and 
displacement of the incident beam of light with the position of each 
measurement pixel. This leads to quantification of the characteristics of 
camera unit. 

10 BRIEF DESCRIPTION OF THE DRAWINGS 
[ 030 ] FIG.l is a schematic diagram illustrating a pin-hole camera model. 

[031] FIG.2 is a schematic diagram for describing distortion on an image 

captured by a camera. 
[032] FIG.3 is a schematic diagram for describing the calibration data. 

[033]i5 FIGS. 4 A and 4B are schematic diagrams conceptually illustrating how 

to generate the calibration information according to the present invention. 
[°34] FIG.5 is a perspective view showing an overall apparatus for generating 

data for calibration information. 
[035] FIG.6 is a flow chart showing steps for generating data for calibration 

20 by a calibration information generator. 
[036] FIG.7 is a diagram illustrating a position of light source relative to a 

camera when the camera is panned and tilted. 
[037] FIG.8 is a flow chart showing the steps of a calibration information 

generator for determining a direction of incident beam of light. 
[038]25 FIG.9 is a diagram illustrating how to search for a peak position of 

lightness. 
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[039] 
[040] 
[041] 5 
[042] 

[043] 

10 

[044] 

[045] 
[046] 

15 

[047] 

20 

[048] 

25 



FIG. 10 is a block diagram illustrating the structure of a position 
measurement apparatus according to the present invention. 

FIG. 11 is a schematic diagram illustrating the contents of a 
calibration table. 

FIG. 12 is a flow chart illustrating the operation of a position 
measurement apparatus according to the present invention. 

FIG. 13 is a schematic diagram describing a method for calculating the 
three dimensional position of an object. 

FIG. 14 is a block diagram illustrating the structure of a position 
measurement apparatus according to the present invention. 

FIG. 15 is a block diagram for a laser unit according to the present 
invention. 

FIG. 16 is a perspective view illustrating illuminated laser beams. 

FIG. 17 is a diagram showing a coordinate system and the definition 
for angles which are used in description of calculation method for a position 
illuminated by a laser beam. 

FIG. 18 is a flow chart showing the steps for operating a position 
measurement apparatus according to the present invention. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Embodiments of the present invention are now described in detail. First, 
description is given to characteristics of a camera which intrinsically conflict 
with an idealized assumption of pin-hole camera that incident beams of light 
cross at a point. This leads to one of causes for distortion of an image captured 
by the camera with a lens system. Subsequently, description is given to 
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calibration information (calibration table) of quantified characteristics of the 
camera, and to a method for generating the calibration information by 
conducting measurement for individual pixels captured by the camera. And 
description is given to an apparatus for detecting a position of an object, which 
can remove distortion of the image. 

Non-pin-hole Characteristic of a Camera 

Referring to FIG.2, description is given to a cause for creating distortion 
on an image captured by a camera with a lens system. FIG.2 is a schematic 
diagram of the camera with the lens system. For convenience sake, it is 
assumed that a lens system of a camera unit C is modeled with a plate glass G 
and a pin hole H is formed by a diaphragm F. An incident beam of light rl, 
which perpendicularly penetrates the plate glass G, forms an image on a pixel 
Rl of an image plane I. Other incident beams of light r2 and r3, which 
diagonally penetrate the plate glass G, form images on pixels R2 and R3 of the 
image plane I, respectively. 

It is known that the camera unit C cannot be qualified as a pin-hole 
camera because lines r22 and r33, which are extensions of the incident beams 
of light r2 and r3 before penetrating the plate glass G, do not intersect with 
the incident beam of light rl at one point. In this way, the incident beam of 
light r3, which is offset by a distance D from an incident beam of light rr, 
forms the image on the pixel R3. 

As described above, the characteristics of a pin-hole camera do not hold 
in a case of camera, which forms an image with incident beams of light 
penetrating a lens system (in this case the plate glass G). A camera with a lens 
system is hereinafter referred to as "non-pin-hole camera". 
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Calibration Data 

Referring to FIG. 3, description is given to calibration data including 
quantified characteristics of a non-pin-hole camera. FIG.3 is a schematic 
diagram illustrating a lens system for description of the calibration data. As 
shown in FIG.3, an incident beam of light R penetrating a lens LS is defined 
by two points. It is understood that if beams of light originating from first and 
second positions PI and P2 of light source form images on the same pixel (not 
shown), the incident beam of light R is meant to be identified as an incident 
beam of light representative of the pixel. 

An optical center O is defined so that the sum of squares of distances 
between all incident beams of light R and the optical center O can take a 
minimum value. On the other hand, a base point K of an incident beam of light 
R is defined as a point where a distance d between the incident beam of light R 
representative of a pixel and the optical center O takes a minimum value. 

In other words, the optical center O(x 0 , y 0 , z 0 )is calculated with 

least-square method applied to the sum of squares of distances d relative to 
the incident beams of light R, which are defined by the first and second 
positions of light sources Pl(x ]9 y } , z,)and P2(x 2 , y 2 , z 2 ). The sum of 
squares is represented by an expression (1). 

d 2 =-(a 2 /b)+c — (1) 

where A, B and C are defined as: 

A =(x 2 -x,X*i -*o)+(y 2 -^iXvi -jO+ta -*iX z i ~ z o) 

B = (x 2 - x ] f + (y 2 -y x f + (z 2 - z, f 
C = (x, -x 0 ) 2 +(y, -y 0 f +(z, -z 0 ) 2 

As described above, the characteristics of a non-pin-hole camera can be 
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quantified by introducing the calibration data, which correlates a direction 
defined by the first and second positions PI and P2 of light source for a 
position of pixel with a displacement from the optical center O to the base 
point K. In this connection, the displacement is represented by a 
three-dimensional vector V D (d x , d yJ d z ). 

It is understood that other types of calibration data may alternatively be 
selected. In the example described above, the vector Vd is defined as a 
displacement originating from the optical center O of a reference point 
perpendicular to an incident beam of light. It may alternatively be possible to 
adopt any reference point as long as it is spatially fixed relative to the camera. 
It may also be possible to select any vector for the vector Vd as long as it 
originates from the reference point to an arbitrary point on the incident beam 
of light. 

Method for Generating Calibration Information (table) 

Referring to FIGS.4A and 4B, a method for generating calibration 
information (table) is described, which correlates data produced by 
quantification of characteristics of a non-pin-hole camera with a pixel. 
FIGS.4A and 4B are schematic diagrams showing the principle of generating 
the calibration information. FIG.4A illustrates conceptually how to measure 
the calibration data for a given incident beam of light by changing pan and tilt 
of the camera. In contrast to this, FIG.4B illustrates a method of measuring 
the calibration data for a fixed camera by changing the incident beam of light. 

As shown in FIG.4A, the calibration information is generated by 
specifying a direction of incident beam of light R imposed on a pixel of the 
camera unit C in the following steps: moving the position of a light source in 
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one direction, one-axis direction defined by the positions PI and P2 of light 
source; determining the incident beam of light R defined by the positions PI 
and P2; and adjusting the pan and tilt of camera unit C in two axes so that the 
beam of light R can impinge on the pixel (measurement pixel). 

[059] 5 As shown in FIG.4B, it may alternatively be possible to specify the 

direction of incident beam of light R which is defined by the positions of PI and 
P2 of light source impinging on the measurement pixel in the following 
manner. The positions PI and P2 are respectively moved in X, Y and Z 
directions (three-axis directions) so that the incident beam of light R can 
10 impinge on the measurement pixel, while the camera unit C is fixed. 

[ 060 ] Introducing one of the methods described above shown in FIG.4A or FIG. 

4B, the calibration table is generated from the incident beam of light R 
specified for the measurement pixel. As shown in FIG.3, the table correlates 
the direction of incident beam of light R and the displacement Vd from the 
15 optical center O to the base point K with the measurement pixel. 

Calibration Information Generator 
[061] Referring to FIG. 5, the structure of a calibration information generator 

is described, to which a method for generating calibration information 
20 according to the present invention is applied so as to prepare it. FIG.5 is a 

perspective view showing the calibration information generator. 
[062] A calibration information generator 100 includes a camera support table 

(rotatable stage) 110, which can support a camera unit C and rotate it in pan 

and tilt directions. The generator 100 also includes a three-dimensional table 
25 (XYZ stage) 120, which can move a point light source L in three-dimensional 

directions, forward-backward, right-left and upward-downward, relative to the 
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camera support table 110. 

The camera support table 110 includes a pan table 111 which is able to 
rotate about a vertical axis on a horizontal plane, and a tilt table 112 mounted 
rotatably to the pan table 111 about a horizontal axis. The camera unit C is 
mounted to the tilt table 112. 

The three-dimensional table 120 has an X-axis rail 121 which is laid out 
in a forward-backward direction on a horizontal plane relative to the camera 
support table 110, and a Y-axis rail 122 which is laid out in a right-left 
direction and able to move in the forward-backward direction on the X-axis rail 
121, and a Z-axis rail 123 which is laid out in a vertical direction and able to 
move in a right-left direction. The point light source L is movably mounted to 
the Z-axis rail 123 in an upward-downward direction. 

The pan table 111 and tilt table 112, which are driven by a driving unit 
such as a pulse motor (not shown), are able to swing the optic axis of camera 
unit C in upward-downward and right-left directions, respectively. A rotational 
angle detector such as a rotary encoder (not shown), which measures the angle 
of optic axis for the camera unit C, is attached to a rotational shaft for each of 
the pan table 111 and tilt table 112. 

The Y-axis rail 122, Z-axis rail 123 and point light source L are able to 
linearly move on their corresponding rails on which they are mounted because 
they are driven by converter mechanisms (not shown), which convert the 
angular movement of pulse motors into linear one. 

It is understood that the driving unit and converter mechanism are 
controlled by a control unit (not shown), and the angle of optic axis for camera 
unit C and the position of point light source L are arranged so that the control 
unit can access to them. 
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Operation of a Calibration Information Generator 
[068] An example of operation of a calibration information generator 100 is 

described with reference to FIG.6, in addition to FIGS.4A, 4B, and 5. FIG.6 is a 
5 flow chart showing steps for generating a calibration table by the calibration 

information generator 100 with the method shown in FIG.4A. 
[069] At a step SI, the calibration information generator 100 sets a point light 

source L at a predetermined position, a position PI of light source, and makes 

rotations for a pan table 111 and a tilt table 112 about individual two axes so 
10 that a beam of light originating from the point light source L located at the 

position PI can impinge on a measurement pixel of a camera unit C, thereby 

obtaining the resultant amounts of pan and tilt. Subsequently at a step S2, the 

generator 100 measures a position relative to the camera unit C for the 

position PI according to the amounts of pan and tilt. 
[070]is At a step S3, the generator 100 makes one-axis movement for a Y-axis 

rail 122 on an X-axis rail in a direction of X, forward or backward, thereby 

shifting the point light source L from the position PI to a position P2. 
[071] At a step S4, the generator 100 obtains another set of amounts of pan 

and tilt for the position P2 in the similar manner as that of the step SI. 
20 Subsequently at a step S5, the generator 100 measures a relative position for 

the position P2. 

[ 07 2 ] It is possible to specify an incident beam of light R impinging on the 

measurement pixel according to the respective relative positions obtained for 
the positions PI and P2 at the steps S2 and S5, respectively. 

[073] 25 Referring to FIG.7 as well as FIGS. 4 A, 4B and 5, a method for 

calculating a position of light source relative to the camera unit C is described. 
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[074] 



[075] 



10 



[076] 



15 



20 



[077] 
[078] 



FIG.7 is a diagram illustrating a position of light source relative to a camera 
when the camera is panned and tilted. 

It is assumed that the camera unit C is mounted on a fixed reference 
point B of the camera support table 110. If the camera support table 110 or the 
camera unit C is rotated by a pan of -0 and a tilt of-0 tilt , a direction of light 

source as viewed from the camera unit C will coincides with one which occurs 
when the light source is rotated about a rotational center Ot of the camera 
support table 110 by 0 and 0 m . 

Assuming that coordinates of the position of a light source before 
rotation are defined as P(x l9 y x , z x ), and amounts of pan and tilt and 
9 tiU , respectively, then a position P R (R xl , R yl , R zX ) of the light source after 
rotation can be represented by an expression (2). 
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In this way, it is possible to specify the incident ray of light impinging 
on a measurement pixel by calculating relative positions for the two positions 
of the light source after the rotation of camera support table 110. Defining 
coordinates of the position PI as(x lf y l9 z pX ) and P2 as(x p2 , y 2 , z p2 ), 

which are translated into relative positions, the incident beam of light is 
specified by an expression (3) with a real number of t. 
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Returning to FIG.6, the description of flow is resumed. 
At a step S6, the calibration information generator 100 judges whether 
or not measurement is completed for a predetermined number of measurement 
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pixels. If it is not completed (No), the flow returns to the step SI and the 
generator 100 continues specifying another incident beam of light for the next 
measurement pixel. If it is completed (Yes), the generatorlOO calculates, at a 
step S7, an optical center according to the directions of incident beams of light 
which are obtained for the predetermined number of measurement pixels. 

At a step S8, the generator 100 generates a calibration table, which 
correlates a direction of incident beam of light and a displacement from the 
reference point with each of the pixels. It is noted that the generator 100 
executes correlation by interpolation according to the data of nearby 
measurement pixels for other pixels which are not subjected to measurement. 
In this connection, it may be preferable to select the positions of measurement 
pixels in advance, taking into account the characteristics of the lens of camera 
unit C. For example, it may be preferable to adopt a position of measurement 
pixel for every five or less pixels in case of a fish-eye lens, or for every five to 
ten pixels in case of other types of lenses. Although the generator 100 
described above as an example generates a calibration table, it may be possible 
for the generator 100 to include a predetermined function such as a function 
for interpolation according to data of nearby measurement pixels. 
Alternatively, it may be possible for the generator 100 to include not only a 
numerical expression to translate a position of pixel into a direction of incident 
beam of light, but also the other numerical expression to translate this position 
of pixel into its displacement. 

Referring to FIG.8 as well as FIG.5, further description is given to how 
the camera unit C is rotated at the steps SI and S4. FIG.8 is a flow chart 
showing the steps by which the generator 100 directs an incident beam of light 
so as to impinge on a position of measurement pixel. 
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[081] As shown in FIG.8, at a step S10, the generator 100 incorporates an 

image captured by the camera unit C. In this connection, it is assumed that 
only the point light source L is on the image, namely a white point on the 
image against a background of black. 

[082] 5 At a step Sll, the generator 100 searches for a position (peak position) 

in which a maximum of the light intensity of white point falls. 

[083] Referring to FIG.9, description is given to how the generator 100 

searches for the peak position. FIG.9 is a graph illustrating the relationship 
between a position of pixel />(..., P n _ 2 , P n , P n+l9 P n+2 , ...) on the 

10 horizontal axis and light intensity I on the vertical axis. The light intensity I 
has a distribution with a peak at a certain pixel. 
[084] A peak position P e where the light intensity I takes the maximum 

value can be obtained by numerical expressions (4) and (5). It is defined that 
I n represents a light intensity at a position of pixel P n ; I n _ x and I n+l 
15 represent light intensities at positions of pixels P n _ x and P n+l next to P n , 
respectively. 
If 

Pe = Pn + (/„ + . " 4-. )/{2(/„ - /„-. )} - (4) 

If 

20 P e = P„ + (/„*■ - /„-, )/{2(/„ - / n+1 )} - (5) 

[085] In this connection, searching of the peak position P e is carried out by 

investigating light intensities of neighboring pixels in both vertical and 
horizontal directions. 

[086] At a step S12, the generator 100 judges if the difference between the 

25 peak position P e searched for at the step Sll and the position of a 

measurement pixel is less than or equal to a predetermined threshold value. If 
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it is (Yes), judging that the peak position P e coincides with the position of this 
measurement pixel, the generator 100 terminates the flow. 
[° 87 ] If the difference is greater than the predetermined threshold value (No), 

at a step S13, the generator 100 converts the difference into angles, thereby 
5 calculating amounts of rotation required for the pan and tilt tables 111 and 
112, respectively. At a step S14, the generator 100 rotates the pan and tilt 
tables 111 and 112 according to the amounts of rotation calculated at the step 
S13, and returns the flow to the step S10, repeating the flow until the peak 
position P e coincides with the position of the measurement pixel. 

[088]io Although the operation of calibration information generator 100 has 

been described, which generates a calibration table with the method shown in 
FIG.4A, it may be possible to generate it with the other method shown in 
FIG.4B. 

[089] In this case, the generator 100 determines the position PI of light source 

15 by moving the position of point light source L in at least one of Y and Z 
directions, positively or negatively so that the maximum light intensity of 
incident beam of light falls in the measurement pixel while the camera unit C 
is fixed. After changing the distance between the point light source L and the 
camera unit C in an X direction, the generator 100 similarly determines the 
20 position P2. In this way, determining the positions of PI and P2 of the light 
source, the generator 100 specifies the direction of incident beam of light R. 
After repeating these steps according to the number of measurement pixels, 
the generator 100 conducts the steps S7 and S8, thereby generating a 
calibration table. 



25 



a. First Embodiment 
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Referring to FIG. 10, description is given to a position measurement 
apparatus. FIG. 10 is a block diagram illustrating the structure of position 
measurement apparatus according to the present invention. A position 
measurement apparatus 1 can measure a three-dimensional position of an 
object OB (marker M) according to an image captured by a camera unit C 
including a set of two cameras CI and C2 (non-pin-hole cameras). The position 
measurement apparatus 1 includes an image input module 10, a pixel position 
module 20, a storage module 30 and a calculation module 40. The marker M, 
which is for indicating a spot for detection, includes a seal with a given color 
and shape, a light emitting diode of infrared beam and the like. 

The image input module 10 incorporates images of the object OB 
captured by the camera unit C. When the image input module 10 sequentially 
incorporates dynamic images captured by the cameras CI and C2, it 
synchronizes them. The image input module 10 has a memory (not shown) 
temporarily storing the captured images, to which the pixel position module 20 
is adapted to have access. 

The pixel position module 20 can detect the position of a pixel, which is 
representative of the object, in each of the images incorporated by the image 
input module 10. In case of a marker M is introduced, the center of gravity for 
an area containing the image of the marker M is considered the position of a 
pixel that is representative of the marker M. The position of the pixel detected 
for each image is sent to the calculation module 40. 

The storage module 30, which is a storage medium such as a memory, 
stores a calibration table 31 correlating the calibration data with each of the 
pixels for each of the cameras CI and C2. Calibration tables 31a and 31b are 
prepared for the cameras CI and C2, respectively. The data of calibration table 
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31 includes the direction of an incident beam of light impinging on each of the 
pixels and the displacement Vd from the reference point (the optical center O) 
to the incident beam of light, as described above with reference to FIG.3. 
[094] As shown in FIG. 11, correlating with a set of coordinates x and y of a 

5 pixel, the calibration table 31 includes the displacement V D {d x , d y , d z ) of an 

incident beam of light relative to the optical center O and its direction in a 
form of angles a and y , which are for specifying the incident beam of light 
impinging on the pixel. 

[095] It may alternatively be possible to adopt a given function or a 

10 transformational expression instead of the calibration table described above. 

[096] The calculation module 40 can calculate the position (three-dimensional 

position) of an object OB (marker M) according to the data of calibration table 
31 which is correlated with the position of a pixel of an image detected by the 
pixel position module 20. 

[097]15 It may be possible to adopt other types of apparatus instead of the 

position measurement apparatus 1 described above. For example, it may be 
possible for the pixel position module 20 to execute matching of images 
captured by the camera unit C in a predetermined size of block so as to detect 
the position of a pixel representative of an object. 

[098]2o It may also be possible to use three or more cameras to measure the 

position of an object instead of the camera unit C including a set of two 
cameras CI and C2. For example, it may be possible to adopt a set of nine 
cameras. In this case, assigning the camera positioned centrally as a reference 
camera, the position of the object can be more accurately measured by 
25 calculating an average of eight positions detected by combinations of the 
reference camera and the remaining eight cameras. 
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[099] A computer program executes a computer so as to operate its processor 

and memory including the storage module 30, so that the image input module 
10, pixel position module 20 and calculation module 40, which are included in 
the position measurement apparatus 1, are brought into operation. 

[100] 5 Referring to FIG.12 in addition to FIG.10, the operation of a position 

measurement apparatus 1 is described. FIG.12 is a flow chart illustrating the 
operation of the apparatus 1. 

[101] At a step S20, an image input module 10 incorporates images captured 

by a camera unit C including two cameras CI and C2. At a step S21, a pixel 
10 position module 20 detects a position of pixel representative of an object OB 
(marker M) in each of the images. 

[ 102] At a step S22, a calculation module 40 reads the calibration data, which 

is correlated with the position of pixel representative of the object OB, from 
one of calibration tables 31a and 31b that are prepared for the cameras CI and 
15 C2, respectively. At a step 23, the calculation module 40 calculates a 
three-dimensional position of the object OB according to the calibration data. 

[103] Referring to FIG.13 in addition to FIG.10, detailed description is given 

to a method for calculating the tree-dimensional position, which is conducted 
at the step S23. FIG.13 is a schematic diagram describing the method for 
20 calculation. 

[104] As shown in FIG.13, a compensated optical center is defined as 

Oi{x x , y x , z,), which is compensated according to a displacement derived 
from the calibration table 31a correlated with the position of pixel that is 
representative of the object OB in the image captured by the camera CI. 
25 Similarly for the image captured by the camera C2, the other compensated 
optical center is defined as 0 2 (x 2 , y 2 , z 2 ), which is compensated according to 
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a displacement derived from the calibration table 31b. 

A direction of incident beam of light, which impinges on the position of 
pixel for the object OB in the image captured by the camera CI, is defined as 
an azimuth a x and an elevation y l9 which are obtained from the calibration 
table 31a. Similarly, the other direction of incident beam of light associated 
with the camera C2 is defined as an azimuth a 2 and an elevation y 2 , which 
are obtained from the calibration table 31b. It is noted that the elevation y 2 is 

not shown in FIG. 13 because it is not used for a calculation of position. 

A position of object m(P x , P y , P 2 ) can be calculated by numerical 

expressions (6), (7) and (8). 

P x = (xj tana, -y l -x 2 tanar 2 +y 2 )/(tana ] +tana 2 ) — (6) 
P y =(P X -xjtana, +y x — (7) 
P,=fc-Xi)t*ari+*i —(8) 

In this way, the position measurement apparatus 1, which employs 
non-pin-hole cameras, can accurately measure the position of object. 

In this connection, it may be possible to apply a position measurement 
apparatus 1 according to the present invention to a movable robot, an 
automobile and the like. For example, introducing the apparatus 1 into the 
movable robot, it may be possible for the robot to walk stably on a floor 
because it can incorporate information on unevenness of the floor by 
conducting measurement for the position of floor with the apparatus 1. 

b. Second Embodiment 

A second embodiment of the present invention is described with 
reference to FIG. 14. FIG. 14 is a block diagram illustrating the structure of a 
position measurement apparatus according to the present invention. The same 
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symbol is used for an item which is the same as that of the first embodiment 
and its description would not be repeated. 

An position measurement apparatus 200 shown in FIG. 14, which has a 
camera unit C capturing an image of a collimated beam of light that originates 
from a laser unit 250 and impinges on an object OB, can determine a 
three-dimensional position of the object OB according to the image. The 
apparatus 200 swings the direction of beam of light upward-downward and 
right-left so that the position of object OB can be determined according to 
three-dimensional measurement conducted at a large number of points. 

The apparatus 200 according to the second embodiment preferably 
includes, the laser unit 250, the camera unit C, an image input module 210, a 
pixel position module 220, a storage module 230 and a calculation module 240. 

As shown in FIG.15, the laser unit 250 includes a laser gun 251, a 
condenser lens 252 and a diffraction gating 253. Brief description is given to 
each item below. The laser gun 251 produces a laser beam LB, the condenser 
lens 252 condenses the laser beam LB produced by the laser gun 251 and the 
diffraction grating 253 disperses the laser beam LB condensed by the 
condenser lens 252. The diffraction grating 253 disperses the laser beam LB in 
a vertical direction. 

After the diffraction grating 253, a diffusion lens 254 is placed for 
diffusing the dispersed laser beams in one direction so as to form a slit of light. 
The diffusion lens 254 includes a cylindrical lens and the like. The diffusion 
lens 254 diffuses each of the laser beams at an angle of 60 deg. 

As shown in FIG. 16, the laser beam LB emitted by the laser unit 250 is 
diffused in a conical shape, reflecting and scattering in a form of a trajectory 
RL of quadratic curve on a plane PL. The reflection and scattering of the laser 
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beam LB indicates a position illuminated by the laser beam LB. The laser 
beam LB is dispersed into five beams, each of which is diffused in an angle of 
60 deg. Each of these laser beams is emitted radially from an origin Ol, which 
is individually defined for each beam. It may be practically preferable to adopt 
another larger number of beams than five described above, which is selected 
for explanation as an example. For example, if an angle B of 32 deg. and an 
angle C of 1.6 deg. between a pair of neighboring beams are selected, the 
number of beams results in twenty one. 

It may be alternatively possible to adopt a rotation mechanism so as to 
rotationally position a spot laser beam in upward ( x) and lateral (y) directions 
instead of a lens system for diffraction and diffusion used in the laser unit 250 
described above. In this case, it may be preferable to arrange the mechanism 
so that the laser beam can be radially emitted from a sole point so as to relieve 
the effort associated with calculation of the position of an object. 

The image input module 210 can incorporate an image of the object OB 
captured by the camera unit C. The image is in a form of color or 
monochromatic multi-gradation image. The image input module 210 has a 
memory (not shown) which temporarily stores images captured by the camera 
unit C, to which the pixel position module 220 makes access. 

The pixel position module 220 can detects the position illuminated by 
the laser beam LB in the image incorporated by the image input module 210, 
thereby determining the position of pixel. More specifically, judging from the 
distribution of lightness on the image, the pixel position module 220 detects a 
pixel which has the peak of lightness. The position of pixel thus detected for 
each of the images is sent to the calculation module 240. 

The storage module 230, which is preferably made of a common storage 
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medium such as a hard disk, stores a calibration table 231 which correlates 
calibration data with each of the pixels of the image captured by the camera 
unit C. As shown in FIG. 11, correlating with a set of coordinates x and y of a 
pixel, the calibration table 231, which is for determining the incident beam of 
5 light impinging on the pixel, includes a displacement V D [d x , d y , d 2 ) relative 
to an optical center O and a direction in a form of angles a and y . As shown 
in FIG. 17, the angle a is an azimuth relative to an X-Z plane of Cartesian 
coordinates (X, Y, Z) whose origin is arranged to coincide with the origin 
Ol of the laser beam LB emitted by the laser unit 250. On the other hand, the 
10 angle y is an elevation relative to an X-Y plane. 

[119] The calculation module 240 calculates a position (three-dimensional 

position) of the object OB according to the data of calibration table 231, which 
is correlated with the position of pixel of each of the images detected by the 
pixel position module 220. 

[120]i5 Referring to FIGS. 17 and 18, the operation of a position measurement 

apparatus 200 is described. FIG. 18 is a flow chart showing the steps for 
operating the apparatus 200. 

[121] At a step ST1, the apparatus 200 illuminates an object OB with a laser 

beam LB. At a step ST2, a camera unit C captures an image of the object OB. 
20 At a step ST3, an image input module 230 incorporates the image captured at 
the step ST2. 

[ 122] At a step ST4, a pixel position module 220 detects a peak of lightness on 

the image which is incorporated by an image input module 210, thereby 
determining a position of pixel which corresponds to the spot illuminated by 
25 the laser beam LB on the object OB. 
[123] At a step ST5, accessing to a calibration table 231, a calculation module 
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240 obtains the calibration data correlated with the position of pixel 
determined at the step ST4. At a step ST6, the calculation module 240 
calculates a camera base position P C {C X , C y , C z ) according to the calibration 

data obtained at the step ST5. 

As shown in FIG. 17, the camera base position Pc is meant to represent 
an origin from which a vector of line-of-sight for the camera unit C starts. If 
the optical center of camera unit C is defined as o(O x , O y , O z )> the camera 

base position Pc is represented by the following numerical expressions. 

C x =O x +d x 
C y =O y +d y 
C z =O z +d z 

In this connection, it may be possible to prepare a calibration table 231 
which correlates the camera base position P C (C X , C y , C z ) instead of the 
displacement V D {d x , d y , d z ) with a position of pixel. 

At a step ST7, the calculation module 240 calculates a position 
illuminated by the laser beam LB according to the camera base position 
P e if! x * C y , C z ), the angles a and ^derived from the calibration table 231 in 
addition to angles co and (f> which define the direction of laser beam LB. If 
the distance of an object is limited, limited laser beams impinge on a given 
area of image. Therefore in this case, it may be possible to determine a laser 
beam according to the position of a pixel, so that the angles co and 0 are 
accordingly determined. If a laser unit is arranged so that it can mechanically 
swing a spot beam, it may be preferable to mechanically or electrically 
measure the direction to which the spot light is directed. In this way, it may be 
possible to incorporate the angles co and <f> , which are synchronized with the 
image, into the calculation module 240, which calculates a position illuminated 
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by the laser beam LB. 

The calculation of a position is conducted as described below. 

As shown in FIG. 17, it is defined that a distance from the origin of the 
laser beam LB to an illumination point P Q (X, Y, Z) is r and the other 
distance from the camera base position Pc of camera unit C to P Q (X, Y, Z) 
is R. 

The illumination point P 0 (X 9 Y, Z) is geometrically related to the 
direction vector of laser beam LB having a distance of r in the following 
numerical expressions (9), (10) and (11). 

X = r cos co — (9) 

Y = rsina) —(10) 
Z = -r cos a> sin (p — (11) 

The relationship between the illumination point P 0 (X, Y, Z) and the 
line-of-sight vector of camera unit C having a distance of R is also represented 
by the following numerical expressions (12), (13) and (14). 

X = Rcosacos r + C x — (12) 

Y = Rsinacosy + C y ™ (13) 
Z = -i?sin r + C z —(14) 

The distances r and R can be derived from these expressions (9) through (14). 

For 0<Cy and 0<co, or Cy <0 and co <0: 

R = (- B + yj(B 2 -4AC))/2A 
For cd = 0: 

R = -C y /sinacosy 
For 0<Cy and co<0, or Cy <0 and 0<co : 

R = (-B-yl(B 2 -4AC))/2A 
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For o = 0: 

r = (r sin a cos y + C y )/sin <s> 

Where A, B and C are defined as follows: 

^4 = cos 2 a cos 2 y + sin 2 y -cos 2 <asin 2 or cos 2 //sin 2 <y 
B-C x cos a cos y - C z sin y - C y cos 2 a> sin « cos y jsm 1 co 
C = C X 2 + C 2 2 - C y 2 cos 2 co /sin 2 <y 

The position measurement apparatus 200 according to the present 
invention, which compensates the non-pin-hole characteristics of the camera 
unit C, can accurately measure the position illuminated by a laser beam. 
Therefore, it may also be possible to measure a three dimensional shape for 
the object OB by conducting calculation of three-dimensional positions for all 
positions illuminated by the laser beam LB. 

In this connection, it may be possible to apply a position measurement 
apparatus 200 according to the present invention to a movable robot, an 
automobile and the like. The apparatus, which is mounted on the movable 
robot so as to detect a floor, can accurately recognize unevenness of the floor, 
thereby allowing the robot to walk stably. 
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